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Abstract 

The current situation regarding evaluation and accountability 
issues in career and employment counselling and the background 
conditions giving rise to the current state of affairs form the 
basis for an evaluation model that can address both formative and 
summative evaluation concerns in career and employment 
counselling. 
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A Framework For Quality Control , Accountability, And Evaluation 
The topic of evaluation in career and employment counselling 
circles is fraught with paradoxes and contradictions. Counsellors 
lament the fact that they are vulnerable in the face of growing 
accountability concerns , but do not see evaluation as a solution. 
A course in evaluation is an optional part of most counsellor 
training programs , however counsellors do very little evaluation 
in their jobs. Accountability is thought to be important , but few 
models exist for helping counsellors become more accountable* 
Most counsellors report receiving an annual performance appraisal, 
and that their work with clients is an important factor in that 
appraisal, however very little first hand observation of their 
work with clients is conducted. What is needed is an evaluation 
framework that is user friendly, has the potential for both 
formative and summative evaluation, and could be used by 
counsellors in trouble shooting their interactions with clients. 

This paper will begin by describing the current state of 
affairs regarding evaluation and accountability issues in career 
and employment counselling, based on a recent survey of career and 
employment counselling in Canada (Conger, Hiebert, & Hong-Farrell, 
1993). The paper then describes some of the background conditions 
giving rise to the current state of affairs. Finally, the focus 
shifts to an elaboration of one model for addressing evaluation 
and accountability concerns in career and employment counselling. 

The Current Situation 
Conger et al. (199?) found that very few counsellors conduct 
any formal evaluation of their work with clients. In some 
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sectors , 40% of counsellors reported never doing any evaluation. 
Less than one-quarter of the counsellors had any sort of form for 
assessing the impact of counselling on clients and less than one- 
fifth did any follow-up* About 35%-45% reported doing informal 
evaluations of their counselling in the session with the client, 
presumably by asking the client if the session was useful. 

Conger et al. (1993) found that attempts at formative 
evaluation were likewise paltry. Few counsellors engage in any 
regular review of the programs they offer clients. When program 
review is conducted, clients are seldom consulted in the process. 
Counsellors do recognize that this presents a problem for they 
report feeling that the attention given to program review, 
evaluation, and improvement is not adequate. However, that 
recognition does not seem to provide sufficient incentive to 
change the current practice. 

Paradoxically, most counsellors report receiving an annual 
performance appraisal. However, their work with clients does not 
appear to be central in their performance appraisal for 
counsellors report receiving very little supervision of their work 
with clients. When supervision is received, it appears to be on 
administrative matters, rather than a counsellor's work with 
clients. Still f 50%-65% of counsellors report that client 
feedback is one of the criteria that go into the performance 
appraisal, but it is not clear what the source of that client 
feedback is. 

Continuing the series of paradoxes, counsellors report 
believing that they offer good service and that their clients are 
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satisfied. In fact, 33%-50% of the counsellors report that they 
think they offer very good or excellent service and 70%-87% think 
their service is as good as or better than the service offered by 
others. However , all counsellors are not sure that counselling is 
meeting their superiors 1 or funders' expectations , and only 50%— 
65% of them feel that their superiors really understand 
counselling . On the average counsellors believe that counselling 
satisfies abut 65%-85% of their superiors' expectations, but in 
some sectors 12% of counsellors report that counselling satisfied 
none of their superiors' expectations. Further, 18%-42% of the 
counsellors could not identify the extent to which their 
superiors 1 expectations were being met. Therefore, while 
counsellors report satisfaction with the work they do, they have 
little objective data to support their belief. 

Background Factors Affecting Evaluation Issues 
It is unfortunate that career and employment counselling have 
drifted from their roots in education (cf. Parsons, 1909) and 
become oriented towards a perspective where evaluating client 
change is problematic. Although most counsellor training includes 
some course work in assessment, often the focus is on intra- 
psychic factors (like personality or temperament), intra- 
individual factors (like personal interests or aptitudes), or on 
vague constructs (like vocational maturity). For years, even 
career exploration has been embedded in psychological tests, 
rather than individual self -awareness, per se. Usually, the 
results of such assessments are expressed in norm-referenced 
terms, rather than criterion-referenced, or individual terms. 
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This may indicate how a client compares to the rest of the 
population, but often does not provide the basis for determining 
client change or progress towards a counselling goal. This sort 
of focus in career and employment counselling makes assessing 
client change problematic, but in a very real sense, career and 
employment counsellors should not be blamed for this state of 
affairs, for they are victims of the training they received. 

A second unfortunate circumstance in the development of 
career and employment counselling has been a pervasive focus on 
process to the exclusion of outcome. Those who need convincing of 
this claim might consider asking a group of counsellors or 
counsellor educators to list say five words that were associated 
with, or describe, counselling. Invariably, the list would 
contain almost exclusively process words. They would be terms 
such as: helping, relationship, facilitation, empowering, 
accepting, supporting, encouraging, and so on. Typically, there 
would be very few outcome words, like client change, problem 
resolution, or getting a job. Seeing counselling as solely, or 
even primarily, a process, and excluding outcome from being part 
of the conceptualization of counselling, makes it very difficult 
to give evaluation a high priority. Moreover, excluding outcome 
from the conceptualization of counselling likely is responsible 
for the prevailing view that assumes attempts to evaluate 
counselling invariably will be intrusive and will interfere with 
counselling (the counselling process). 

A major tenet in this paper is that there is a need to build 
a framework that includes outcome and process as equal partners in 
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the counselling enterprise: Process without outcome is not 
counselling , any more than is outcome without process* To extend 
the argument , the belief underlying this paper is that it is 
important to build a framework for counselling that incorporates 
evaluation as a regular part of the counselling process , that 
centers evaluation in the client's experience (rather than the 
frame of reference of the counsellor or the test), and views 
outcomes and process as inextricably entwined, both being integral 
parts of what counselling is all about. 

An Evaluation Framework 
Any attempt to build a framework for the evaluation of career 
and employment counselling must begin by examining the question 
"What are the legitimate processes and outcomes of counselling?" 
Ideally, the answer to this question will result from negotiations 
between the various stakeholder groups involved: clients, 
counsellors, supervisors, managers, and fund?is. In these 
negotiations, counsellors need to make sure that the processes and 
outcomes so identified are "achievable" so that when the criteria 
for accountability are identified counsellors commit to a 
"deliverable" goal. This should involve discussions between 
supervisors, managers, funders, and counsellors to create a shared 
view of the legitimate processes and outcomes associated with 
counselling and to make sure that everyone's expectations pertain 
to a legitimate domain for counselling. Having reached agreement 
on legitimate domain and reasonable expectations, the focus can 
then shift to identifying the legitimate processes and outcomes 
associated with counselling. 
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Counselling Outcomes 

Recently, some authors (e.g., Killeen, White, & Watts, 1993) 
have pointed out that although people often expect counselling to 
produce results that have a direct socio-economic the impact, the 
more legitimate outcomes of counselling are the learning outcomes* 
Killeen et al., (1993) claim that it is inappropriate to judge the 
success of counselling on factors over which counselling has no 
control. Instead, it is more appropriate to .evaluate counselling 
on the learning outcomes which are the precursors of socio- 
economic change. Thus, when people learn the skills involved in 
overcoming employment barriers like finances, family conflict, and 
lack of support, counselling should be considered successful, even 
if the immediate result is not job placement. Similarly, when 
people learn self-management skills that have been shown to have a 
positive impact on motivation (e.g., setting realistic, 
achievable, and observable goals, self -monitoring progress towards 
those goals, reward oneself for goal-directed thoughts and 
actions, see Bandura 1977, 1980; Kanfer, 1980), counselling should 
be judged successful, because motivation (a precursor) can be 
shown to have a positive impact on job placement rates, job 
productivity, or other similar socio-economic variables. 

In a similar vein, as counselling theory continues to embrace 
a systems perspective, counselling should adopt more of a "system 
management" role, focusing on "navigation through" the system, 
rather than "transition from" it (Killeen et al. , 1993, p. 78). 
Thus, the interpersonal skills, leadership skills, and teamwork 
skills that are part of the Employ ability Skills Profile, 
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published recently by the Conference Board of Canada (1992) become 
legitimate outcomes for counselling because they have been 
identified by employers as skills important for job success and 
job satisfaction. The important point underlying this discussion 
is that we need to conceptualize the goals of counselling in terms 
of the legitimate outcomes that can be achieved by counsellors and 
clients working together* 
Counselling Process 

For some time now, counselling has been shackled by an over- 
reliance on the individual interview as the primary method of 
operating. The vast majority of clients seeking in career and 
employment counselling in Canada are serviced with individual 
interviews. Conger et al. (1993) found that upwards of 75% of 
clients are seen in individual counselling sessions. If the 
legitimate outcomes of counselling are seen as learning outcomes 
rather than intra-psychic or socio-economic, then it becomes 
appropriate to explore more diverse ways for servicing client 
needs . 

The literature pertaining to individual and small group 
instruction and experiential learning contains many procedures 
that have an established track record of success. These methods 
includes coaching, skill practice (complete with feedback), skill 
modeling, home work, group discussion, workshops, and didactic 
instruction, all of which should be considered as part of the 
counselling process. (It is important to note that while some of 
the above concepts are present in the counselling literature, 
e.g., homework, they typically are not considered an integral part 
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of counselling. To illustrate, methods for determining, 
assigning, monitoring, and revising homework are not included in 
the skill training that counsellors receive.) Very few people are 
"talked into" achieving learning outcomes. What is more typically 
successful is a variety input, based on the learner's needs, 
relevant to the learner's frame of reference, graduated and 
sequenced so that the learner can master the various learning 
components, and embedded in a total learning experience that will 
help learners integrate the new knowledge, skills, and attitudes 
into their existing experiential framework. 
A Model For Evaluation 

Figure 1 presents one possible framework for approaching the 
evaluation of career and employment counselling that incorporates 
the suggestions given above. In Figure 1, process and outcome are 
seen as equal partners in the counselling enterprise. Counselling 
process has two primary components, namely what the counsellor is 
doing and what the client is doing. Ideally, counsellor action 
will be guided by client factors such as: client goals, preferred 
ways of operating, and past experience. Counsellor actions in 
turn will influence client responding and the counsellor will 
adjust his or her subsequent actions based on those client 
responses. Thus counsellor and client process are seen as 
intertwined in a reciprocal relationship. Ideally, counsellor and 
client processes will complement one another, but in order to 
evaluate the processes leading to counselling outcomes, evaluation 
procedures need to allow for the differential effects of 
counsellor and client actions. 
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Insert Figure 1 About Here 

When counselling is successful, counsellor and client 
processes will lead to client learning outcomes, i.e., changes in 
knowledge, skills, attitudes, etc. Those learning outcomes in 
turn will impact such things as motivation, self -awareness, 
capacity for self-direction, personal agency, and other precursors 
of more global change, or on socio-economic variables directly, or 
on aspects of the client presenting problem. 

The model presented in Figure 1 can be used for both 
formative and summative evaluation and as a guide for trouble 
shooting problems that arise. For example, if the socio-economic 
outcomes are not achieved or if there is inadequate change in the 
client presenting problem, it becomes appropriate to examine 
whether or not the learning outcomes have been achieved. If 
adequate client learning did not occur, then it is important to 
examine whether the counsellor implemented the intervention in an 
appropriate manner and how closely the client followed the 
intervention plan. 

To illustrate how the components in the model are inter- 
related, consider the example of a client who attends a Job 
Finding Club and does not find a job. In order to trouble shoot 
the reasons for this unexpected result, it is important to have 
evidence pertaining to each component in the model. Regarding 
skill acquisition, if the client skill levels are high then as 
long as the client stays motivated to use the skills a job offer 
likely will follow. 
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On the other hand, low client skill levels across the skill 
domains addressed in the Job Finding Club could explain why there 
was no job offer. If it appears that very little client learning 
orourred, then it is important to examine the process variables. 
If the client was not engaged in the program, it is no surprise 
that little learning resulted and consequently, little change in 
the presenting problem. In such cases, the counsellor would need 
to find ways to make the intervention more engaging or meaningful 
to the client so that a higher degree of compliance to the 
intervention could be obtained. On the other hand, if the client 
followed the program but still did not achieve sufficient skill 
mastery, they perhaps more practice is necessary or perhaps a 
different program, more suited to the client's style or past 
experience, is warranted. 

Counsellor fidelity to the intervention also is important 
when trouble shooting an unanticipated result. Continuing with 
the example from the Job Finding Club, it may be that the 
counsellor was not following standard Job Finding Club procedures, 
which would explain the unsuccessful result. For example, if the 
counsellor was primarily lecturing group members, of if the focus 
was primarily discussion rather than modeling, skill practice, and 
positive reinforcement, then little client skill acquisition would 
be expected. It is only when the counsellor follows a reasonable 
approximation of the intervention that we can expect the result 
typically associated with that intervention. 

A key factor in this evaluation model is the explicit linking 
of process and outcome. Further research will be needed to make 
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explicit the link between learning outcomes , precursors , and 
socio-economic outcomes , but a few examples can already be 
provided. One example mentioned earlier is the connection between 
basic self-management skills (e.g., goal setting, self -monitoring, 
and self -reinforcement) , motivation, and job search behaviour. 
Another example is provided by the relationship between learning 
relaxation (skill), reducing stress (precursor), and success in 
job interviews (socio-economic). Killeen et al. (1993) point out 
that assessing learning outcomes is technically simple and quite 
cost effective. However, they admit that the economic outcomes 
are more difficult to evaluate, from both a technical and cost 
perspective. Therefore, once the link between learning and 
economic outcomes is established, the learning outcomes can be 
routinely evaluated, likely at a local level, while the economic 
outcomes may be evaluated less frequently and perhaps on a larger 
scale . 

Evaluation Methods 

One characteristic inherent in this model will be a far 
greater role for informal evaluation procedures. In the past, 
formal standardized assessment techniques have predominated in 
career and employment counselling circles, while informal methods 
have been disparaged. However, the rich heritage of informal and 
practitioner generated evaluation procedures, that have been part 
and parcel of other learning environments e.g., schools, will be 
invaluable to draw on when developing methods for monitoring (and 
evaluating) counsellor and client process and for assessing 
learning outcomes and precursors. These methods will include 



14 



Evaluation Framework/ 14 

simple check lists of things such ass client home work completion, 
number of job interviews attended, level of subjective stress 
during an interview, motivation swings during the day. (See 
Figures 2 for examples.) They also might include specially 
designed client progress forms that document referral criteria and 
summarize and tabulate client progress towards counselling goals. 
(See Figure 3. ) 



Insert Figures 2 and 3 About Here 



The recent descriptions of performance assessments in the 
education literature (cf. Wolf, Bixby, Glenn, & Gardner, 1991) 
have applicability to counselling interactions. Using performance 
assessments involves using the same activities that are part of 
skill training in the assessment of skill mastery, thus ensuring 
that assessment involves a meaningful task that is not intrusive 
to the learning environment. Such procedures show great promise 
for evaluating client learning outcomes in counselling. 

Increasingly, some organizations are finding that they are 
not able to meet the total array of client needs and are beginning 
to "contract out" services to smaller, more specialized agencies. 
However, often it is difficult to predict the success of such 
referrals and to monitor how well the contracted agency is filling 
the referring organization's expectations. The check lists 
provided in a recently published Quality Service Workbook (Riddle, 
1994) provide an example of how organizations can systematically 
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approach the selection and evaluation of referral sources or 
examine their own quality assurance practices. 

Such informal assessment and monitoring procedures are 
relatively easy to use, unobtrusive , and do not detract 
appreciably from counselling process. They offer great potential 
for making more explicit the nature of the legitimate outcomes of 
counselling and for establishing the links between those outcomes 
and the "hard data results" that most planners and funders 
currently are enamored with. 

Summary 

The main argument in this paper is that for counselling to 
survive in times of reducing resources and increasing 
accountability , an new approach for the role of evaluation needs 
to be developed. This new approach will need to place evaluation 
as a high priority in the counselling endeavour. Evaluation will 
need to become as much a part of counselling as the interpersonal 
skills that form the bulk of counsellor training programs. 

Killeen et al. (1993) see a ner.d to develop an "evaluation 
culture" , where evaluation becomes a routine feature of 
counselling practice. The focus in this evaluation culture will 
need to be on demonstrating effectiveness , rather than counting 
inputs , or even counting outcomes that may or may not be related 
to the inputs or the counselling process. From this perspective , 
the focus becomes one of demonstrating effectiveness, rather than 
user reports of general satisfaction. 

With the emphasis in counselling on systems approaches , there 
will be an increasing need to view the outcomes of counselling in 
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terms of "system management" rather than "transition from" the 
system. The more counselling adopts a systems perspective f the 
less realistic it is to evaluate it in terms of "post- 
transitional" measures. Thus,, there will be a greater emphasis on 
assessing the learning outcomes that lead to successful navigation 
through the system and linking those learning outcomes to the 
post- trans it ion measures that currently predominate the evaluation 
field. 

It was mentioned earlier that the learning outcomes of 
counselling can be assessed locally and can be tapped in a way 
that is technically simple and cost effective. This would allow 
for their assessment in a routine and unobtrusive manner. The . 
economic outcomes provide a greater challenge for they are more 
difficult from both a technical and cost perspective. Therefore, 
the development of economic assessments , and the establishment of 
the links between the learning outcomes and the economic outcomes , 
likely is the job of national action bodies and large research 
studies. The potential pay off, to both the profession and the 
clients it serves , is well worth the effort. 
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Figure Caption 
Figure 1. A Model For Evaluating Career Counselling 
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Figure Caption 

Figure 2. Sample Checklist For Monitoring Client Process 
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Job Interview Skills Practice Log Sheet 

Use the following form to keep track of the job interviews you have attended. 
Use the following scales to chart your reactions. 





Interviewer rating 


l=rotten 2=OK 3=really friendly 




Your skills 


1 -terrible 2=not too good 3=OK 4=mostly good 5=excellent 


How you felt the interview went 


l=terrible 5=OK- 10=great 



Date 


Interviewer Name 


Interviewer 
rating 


My skills 


How I felt 





















































Level of Motivation 

Place a mark on the chart below to show how motivated you feel to look for work. 

Use the five-scale below. 

5 = very motivated-it's the most important thing for me to do today 

4 = very motivated, but something might come up to interfere 

3 = quite a bit motivated, I think I will end up looking for some leads 

2 = I sort of care and I might get around to looking for work today 

1 = If I run out of other things to do, 111 try looking for some leads 

0 = I couldn't care less if I ever looked for work 
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Figure Caption 

Figure 3. Sample Checklist For Monitoring Client Outcomes 
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Group Job Finding Clubs Client Tracking Check List (to be completed for each participant) 



Referral Criteria 


Criteria met 
Y ? N 


Contractor Expectations 


Expectations for 
Client 


Evaluation 


wants full time employment 








Return 1 copy of this report 


Client Process 


can handle work in groups 








for each client to the project 


Attendance 




receptive to structured 
approach 








manager as part of the 

1 month follow-up report j 


Network Telephone 
Practice 




English competency 










Resum6 




pmotionflllv ^taHlp 








^iiem jt rocess 


Video interview 
practice 




medical condition stable 








refers to how well the 


Application Forms 




not booked for surgery 








worker engaged in the 


Cover Letter 




no medical investigation 








training process for each 


Job Leads 




no medication interference 









element 


Job Offer 
Evaluation Form 




not in active appeal process 










Client Skills 


full 1 1 tv^ o o/**rt/"Y/~\l pnnantila/i 

no iuu lime scnooi scneuuiec 
within 4 mo 








Client Skills 


Telephone skills 
* Network calls 




no suDsuince douse 








reiers to now wen ine 


• job leads 




not emotional lv challenged 

vlliUUUllul A j VllUllvllKvU 








client mastered the skills 
taught in the 


| Informational 
interview skills 




Case Mgr approval (if 
previously enrolled in JFC) 








Job Finding Club 


Job interview 
skills 












Client Outcomes 


Client Outcomes 










refers to the products that 


j Network chart 




presession completed 








workers produce during 


calling cards 




has defused anger, is ready to 
proceed 








the Job rinding Club 


Sample 

application forms 




has identified transferable 

olri 11c 








These should be part of 


Resumd 




has clear job objective 








each client's file 


j Cover letter 




commit to 5-6 hrs/day, 3 wks 










1 Job offer 












j Job accepted 


N Y 



In evaluating the outcomes, please use a two-step process. 

(A) decide on whether the outcome is acceptable or unacceptable, then 

(B) assign the appropriate rating: 

(1) if the outcome is just terrible, 

(2) if the outcome really is not acceptable, but almost there, 

(3) if the outcome is barely acceptable (but still OK otherwise it would be 1 or 2), 
(5) if the outcome is outstanding, and 

q (4) if it is somewhere between minimally acceptable and outstanding. £j Q 

ERXCaphically, the scale looks like this: 



[Unacceptable ! Acceptable 

I h- H i H 

0 1 ; 2 3 4 



